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ABSTRACT 



In 1983, California's Los Rios Community College District 
developed the Student Flow Research Model, designed to bring together data on 
community characteristics, student characteristics, student and program 
performance, and former students' outcomes to answer basic questions related 
to institutional effectiveness. Over the years, the District has updated the 
model to include new data base capabilities, renaming it the Collegiate 
Yearly Accountability (CYA) model. The CYA contains an Enrollment Potential 
Projection Model (copyright) that determines the potential for increased 
enrollments and likely effects of different scenarios on enrollment. Another 
improvement has been the combination of student and course master files, 
which allows the school to determine who their students are and what courses 
they are taking. One of the most important developments of the CYA was the 
Student Performance Data Base, which links students’ application, 
demographic, and grading data in order to produce computerized transcript 
analyses, course classifications, and other data. Finally, a new Student 
Follow-up Data Base includes employment information, ratings of college 
courses, program-specific summaries, and data from skills- specif ic 
questionnaires for former students. Through this improved model, the District 
will be able to move from compliance and accountability to true analyses of 
institutional effectiveness. Contains 11 references. (YKH) 
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FROM ACCOUNTABILITY TO EFFECTIVENESS: 
THE STUDENT FLOW MODEL TEN YEARS LATER 



THE ERA OF ACCOUNTABILITY 

A Nation at Risk. Involvement in Learning. To Reclaim a Legacy. Integrity in the College 
Curriculum. The Neglected Majority. Time for Results. Since 1983 a steady stream of these and 
other national reports have focused on the need for educational reform. Calls for higher standards 
and tougher performance criteria have trumpeted from both within and without the educational 
establishment. Something is wrong with American education, the critics charge, and institutional 
defenders have been hard pressed to meet the criticisms with anything more substantial than rhetoric. 
Faced with data that seem to show low retention rates, increased numbers of remedial courses, fewer 
transfer students, longer time to degree, and fewer course and program completions, educators are 
being asked to account to a variety of audiences as to how they are spending public monies and 
whether what they are doing is working (Coffey, 1989). 

The responses of state legislatures across the country to the calls for educational reform have resulted 
in a plethora of commission reports, resolutions, regulations and new laws concerning institutional 
effectiveness. Reforms in access, admissions, remediation, curriculum, and graduation standards have 
surfaced in many states, with concomitant reporting requirements. Almost everywhere, these 
measures represent the states' desire to improve educational quality by causing institutions to become 
more accountable for their outcomes (Moore, 1986). 

The move toward increased accountability at the state level has also introduced the potential for — and 
the reality of — performance-based funding of educational institutions as a logical corollary. 
Competitive incentive grants, withholding of percentages of state appropriations, increased 
percentages of funding based on meeting performance criteria, are all in existence in various states 
across the country. Such performance-based systems certainly put teeth into state level mandates for 
accountability, but also raise crucial questions of who is to determine the evaluation criteria and 
appropriate performance measures. Whether it be legislative staff, state board members, or 
institutionally-based task forces, wise choices in evaluation and performance measures are essential 
to real educational improvement — and potentially even to institutional survival. 

In addition to the state and federal mandates for increased accountability, the regional accreditation 
agencies have instituted their own calls for more accountability through new accrediting standards. 
Beginning in 1984 with the Commission on Colleges of the Southern Association of Colleges and 
Schools, the regional agencies have written standards that established the evaluation of effectiveness 
as a major criterion for accreditation. In 1988, the Western Association of Schools and Colleges 
adopted its standard on "Institutional Effectiveness" that calls for each institution to initiate a research 
program that assesses student skills, performance and outcomes, and links that information to the 
institution's planning process (WASC, 1988, p. 16). 



The California Picture 



In California, the call for increased accountability for the state's community colleges and its linkage 
to institutional research is perhaps nowhere better stated than by the Commission for the Review of 
the Master Plan for Higher Education in its study The Challenge of Change: A Reassessment of the 
California Community Colleges (1986, p. 12): 

Institutional research is essential to determine which types of programs work best 
with which students under what circumstances, and to ensure the wisest use of public 
funds in meeting student and community needs. There is relatively little statewide 
institutional research available to evaluate the effectiveness of the Community 
College transfer, vocational, or remedial programs, which are of particular concern 
to this Commission. If these programs are to be implemented successfully and cost- 
effectively, they must be accompanied by research and evaluation from the start, to 
strengthen these programs as they develop as well as to evaluate their ultimate merit. 
Significant additional funds will be needed for this research. 

To respond to the challenge of accountability, institutions have begun to identify and implement 
research and data collection models that will help them answer both state and local questions about 
institutional effectiveness. What has become apparent over the past several years, however, is that 
there is a significant difference between accountability — which has come to be focused on data 
elements and numerical measures (what used to be called "compliance") — and real institutional 
effectiveness. With the advent of California's community college reform bill (generally known as 
"AB 1725"), and the passage of legislation detailing the state's new accountability system for the 
community colleges, colleges across the state have added research offices to help meet the demands 
for accountability data to be entered into the new statewide Management Information System (see 
Coffey and Hamre, 1988). There it will be analyzed by the State Chancellor's Office staff to help 
answer the various state agency and public questions about how well the community colleges are 
responding to one or another mission or mandate. Clearly, the era of accountability is upon us. The 
remainder of this paper will describe how one district, often noted for its successes in proving what 
it's doing is working, has moved beyond accountability and built a real model for institutional 
effectiveness. 

IN THE BEGINNING: THE STUDENT FLOW RESEARCH MODEL 

In 1983, considerably before the current emphasis on accountability, the Los Rios Community 
College District began to build what would later come to be known as the "Student Flow Research 
Model." Given the district's major emphasis on using research results for planning and decision- 
making, we sought a model that would bring together information in four critical areas: 
(1) Community: characteristics of the district's service area population; (2) Enrolled Students: 
characteristics of the entering student population; (3) College Experience: evaluation of student 
performance, programs and services; and (4) Former Student Follow-Up: follow-up of transfer and 
occupational graduates, and nonretuming students (Coffey, 1987). (See Figure 1.) 
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Rather than focus on data to be collected, we intended that the research produced in these four areas 
would answer a series of seven key questions, questions we thought any district or college should 
be able to answer: 

1 . What is our community like and who are our potential students? 

2. Who are our enrolled students? Do they differ by college? Do they reflect the 
community at large? 

3. What kinds of preparation do our students bring to our institutions? Are they 
prepared for our college-level classes or do they need remediation? 

4. What are the educational goals of our students and do these goals differ by age, sex, 
ethnicity, work status, or economic level? 

5. How well are we meeting our students' needs? Is what we're doing working and how 
do we know? 

6. What happens to our students once they leave? Are they successful as transfers to 
four-year institutions? In finding jobs? In improving skills and potential if currently 
employed? 

7. Finally, how can we improve what we're doing? 

While we had originally thought that these key questions were almost self-evident, the considerable 
interest in the model and questions from California and across the country indicated that the concept 
of building a research model around student and institutional effectiveness questions was quite new. 

For the Community information, we developed district and college service area demographics, based 
on the most recent Federal Census, and "geocoded" to our service areas based on a ZIP code analysis 
of our enrolled students. Labor market information from a variety of sources is included in the 
Community information, as is economic data about particular communities. For the Enrolled 
Students information, we largely use student demographic, goal and financial aid status data from 
the application data base, along with course and unit load information from the course and program 
data base. 

The College Experience area was the one in which we found the least information available when 
we designed the Student Flow Research Model, because the studies of student preparation, 
persistence and performance were in their formative stages. We knew even then, however, that we 
wanted to do internal evaluations of our programs and services according to the "three E's": equity, 
efficiency and effectiveness. To do truly effective program and service evaluations, we knew we 
needed some means of tying student demographic and goal data to course-taking patterns and 
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transcripts — and in the early 1980s, without the kinds of relational data bases that are now in use, 
we had no way to do it. 

Finally, the information on our Former Students was becoming available, based on our Measures 
of Progress studies (Lee, 1987, 1991; Beachler, 1993, 1995), which used survey research to follow 
what had happened to our students once they had left our colleges. Indeed, our research in that area 
was used in pivotal legislative discussions about the supposed high community college drop-out rate, 
when we were able to document that many of our students came for job upgrading and retraining 
opportunities that required only particular courses rather than degrees; in short, we documented the 
phenomenon of the "drop-in" student. 

Over the past thirteen years, we have fine-tuned the Student Flow Research Model and kept our eyes 
on the increasing importance of those seven key questions. And we were able to add, through the 
addition of some unique data base developments, the ability to answer the questions of what was 
happening to our students while they were in our institutions. The next section of this paper will 
focus on our 1990s' version of our Student Flow Research Model and its development into the 
Collegiate Yearly Accountability Model — or our CYA model, for short (and yes, the allusion to the 
more common meaning of those initials is intended). 

THE NEW MODEL AND WHAT MAKES IT WORK 

Figure 2 illustrates the new data base developments that make the model work. In the Community 
information area, at the district level we added the Enrollment Potential Projection Model® or 
EPPM®, an entirely unique computerized enrollment projection model that was approved by the state 
Department of Finance in 1990 for use in projecting the enrollments for Folsom Lake College, the 
district's proposed new college (usually, only DOF can do such projections). In the Enrolled 
Students information area, we combined the demographic and application data from the original 
model (our "student master files") with course and section data (our "master schedule files"). This 
has enabled us to do computerized program review analyses that include three-year demographic, 
enrollment, credit load, and day/evening trend information. In the College Experience area, we 
constructed an entirely new data base the Student Performance Data Base or SPDB — that links the 
student demographic and course data with performance data (transcripts) in a relational data base, 
allowing us to do computerized transcript analysis for any population of students quickly and 
accurately. Finally, in the area of Former Student Follow-Up, we have enhanced our extensive 
student follow-up survey data base through the ability to compile the results not only at the district 
or college level, but also at the program level for use by faculty doing program planning and review. 
The next sections will cover each of these areas in greater detail. 

The Enrollment Potential Projection Model 

The Los Rios Community College District, as part of its new strategic planning process, began in 
early 1991 to update and enhance the enrollment projection model first developed by Dr. Marc Hall 
for the 1988 facilities project. The model developed by the district's Office of Planning and 
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Research was called the Enrollment Potential Projection Model (EPPM) and was quite different from 
the original, due to its capability of producing alternative projection scenarios. This is particularly 
important in a multi-college district with a variety of outreach center locations. We offer a complete 
description of the model here for districts considering adding new colleges or off-campus centers, 
since the model was critical to the Los Rios district's success in gaining approval for a fourth college. 
The EPPM provides the district and colleges with capabilities crucial to comprehensive and cost- 
effective planning: 

(1) The ability to project the potential for increased enrollments in the district and at the colleges 
based on the most accurate data available; 

(2) The ability to project the results of a variety of alternative planning scenarios, including 
differences in community area growth rates and college participation rates, and 

(3) The ability to project the potential impacts on district enrollments of proposed new colleges 
or centers. 

The EPPM was designed to help the district and colleges answer the key planning questions facing 
any district as it examines alternatives to handle projected growth: 

• How many students will we need to serve in the coming years? 

• From which community areas of the district's service region will these students come? 

• What impacts might the differential rates of growth throughout the district s service area have 
on projected district and college enrollments? 

• What impacts might target marketing to particular areas have on potential district and college 
enrollments? 

• What are the available options for meeting the increased demand, and what are the potential 
impacts of each option on the current colleges and centers? 

• Are there enrollment limits that we wish to set for current or projected colleges or centers, 
and what could be the potential impacts of such limits? 

The Enrollment Potential Projection Model projected potential enrollments for the Los Rios district 
and the individual colleges based on current enrollment participation rates, using actual and 
forecasted population projections based on 1990 Census Data and governmental population 
projections. Figure 3 on the next page illustrates the basic components of the model. 
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Enrollment Potential Projection Model 





Areas 



The forecasted population projections for Sacramento and Yolo County were provided by the 
Sacramento Area Council of Governments (SACOG). El Dorado County population projections 
were also supplied by SACOG, but were derived from Department of Finance projections. 

The EPPM divided the Los Rios district into 24 individual community areas. Population projections 
for each of the community areas were supplied by SACOG for 1990, 1995 and the year 2000. 
Actual 1990 enrollments were derived through analysis of the district's Fall 1990 First Census 
Student Master Files. Enrollments were identified for each community area, and broken out by 
college (including the Placerville Center). 

Participation rates of students from each of the community areas were derived by the following 
formula: 



1990 Fall Student Enrollment 
from Community Area 

= Participation Rate 

1990 Population 
from Community Area 



Projected enrollments for any community area for the years 1995 and 2000 were computed by 
applying the following formula: 



1990 Participation Rate 
by Community Area 



Population of 
X Community Area in 
Year of Projection 



Estimate of Student 
Enrollment in Year of 
Projection by 
Community Area 



Individual college enrollments were computed by calculating attendance distributions by college 
using the following formula: 



Fall 1990 Total College 
Student Enrollment from 
Community Area 

Fall 1990 Total District 
Student Enrollment from 
Community Area 



Estimate of 

Ratio for w Student Enrollment 

Each College * in year of Projection 

by Community Area 



Estimate of 
Student Enrollment 
by College by Area 
in Year of 
Projection 



There were five original versions of the Enrollment Potential Projection Model. Model I, the most 
basic model, was patterned after the model developed by Dr. Marc Hall ( Los Rios Community 
College District Student Enrollment Analysis and Projections, Report II; Hall and Coffey, 1988). 
The first page of the model was essentially the "data page" where the parameters were set. It showed 
the base population projections for each community area; the 1990 student population that attended 
from each community; the actual 1990 student participation rate (the percentage the student 
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population is of each community area); the 1995 projected community area populations; the resulting 
projected 1995 student population (based on holding the actual 1990 participation rate steady), the 
year 2000 projected community area populations; and the resulting projected year 2000 district 
student population (again, based on the 1990 participation rate). 

Summary reports for 1995 and the year 2000 were provided on the following pages of the model 
showing projected potential enrollment distributions for the individual community areas for each of 
the three current colleges in the district, and for the Placerville Center, for 1995 and 2000. Totals 
printed at the bottom of each page showed district projected potential enrollments. 

Model II built on the foundation of the first model, but allowed for user-adjustable participation rates 
for each community area Since participation rates can be increased by proximity to a center or 
college, as well as by marketing, the ability to adjust the participation rate was an important part of 
the model. The same summary reports for 1995 and the year 2000 were also included in this model. 

Model III, a more complex model, also provided for user-adjustable participation rates by 
community area, but in addition allowed for adjustable population multipliers for each community 
area. This permitted adjustment of the community area populations if more current population 
projections were available or other factors suggested changes. In addition. Model III allowed the 
user to set enrollment limits for each college, then permitted the "excess" enrollments to be allocated 
to a "new" college if forecasted enrollments for the current college exceeded the user-defined limits. 
The same summary reports for 1995 and the year 2000 were also included in this model. 

A crucial component of the EPPM was the ability to reflect the enrollment limits established for each 
of the district's current colleges and the Placerville Center, so that "excess" enrollments could be 
applied to the "new" campus in the model. Model IV provided for setting of college enrollment 
limits. The Chancellor and the college presidents agreed upon the college enrollment limits to be 
used in the EPPM. These limits were established after consideration of State facility usage 
standards, maximum build-out for each of the current campuses based on the district s 1 99 1 Capital 
Outlay Plan, and the ability to meet recognized and generally accepted levels of service. American 
River College's enrollment limit was set at 21,000; Cosumnes River College was set at 10,000; the 
Placerville Center was set at 3,000; and Sacramento City College was set at 18,000. The EPPM was 
modified to reflect the mutually agreed-upon enrollment limits, and then was run to produce the 
"Model IV" baseline projection. 

Model V, the model ultimately used and approved by the Department of Finance as the formal 
projection for the new Folsom Lake College enrollments, added one additional feature to Model IV. 
a "Special Report" that identified the proportions of people most likely to attend the new college 
from each community area, based on proximity, on analysis of student enrollment by ZIP code, and 
on responses to a community survey. While Model IV projected all the potential new students 
coming to the district, the Model V Special Report recognized that not all the new students from 
throughout the district would attend the new campus, and instead reflected those most likely to 
attend. 
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The EPPM provided a unique means of answering the key questions of what our various community 
areas were like and what level of potential future enrollments we would need to serve. The best 
example of the effectiveness of this new research model was its approval by the state Department 
of Finance, the state Chancellor's Office, and the California Postsecondary Education Commission 
as the basis for the formal enrollment projections for the Los Rios district's fourth college. The 
model was highly flexible and permitted us to analyze any number of alternative scenarios, based 
on adjustable participation rates, variable population multipliers, different college enrollment limits, 
and the likelihood of people from different community areas attending each of our colleges. The 
EPPM has generated a great deal of interest from other community college districts around the state 
as we all seek to understand how we can plan to meet the increasing enrollment demands in face of 
continuing fiscal stringency. 

Combining Student and Course Master Files 

In the area of Enrolled Students mentioned above, we have now combined our student application 
and demographic files with our master schedule files, in order to be able to analyze historical student 
enrollment data and its relationship to changes in course-taking patterns. Figure 4 on the next page 
illustrates the linkages of these files and the kinds of information available. 

Our Student Master Files (SMF) contain student application and demographic information collected 
during the registration process. Information such as educational goal, full- or part-time status and 
major are some of the application items that are collected. Demographic information includes such 
elements as age, gender and ethnicity which are also collected during the application registration 
process. 

Our Master Schedule Files (MSF) contain student course enrollment information collected from the 
students during the registration process. The actual class schedules, which include course, section, 
day and time and instructor information, are stored in these files. 

All information collected from students during the registration process is stored on the district s 
mainframe computer(s). The district's research department routinely extracts the information from 
the Student Master Files and Master Schedule Files stored on the district's mainframes at the end of 
Fourth Week Census. These files are supplied to the college via the Internet. 

The ability to link these two types of quite extensive mainframe files using a PC database and 
statistical software package thus became our first look into how student enrollments both affect and 
reflect course planning decisions. At the college level, having the three-year history of key program 
trends relieved the faculty from having to search for and analyze such information as they had done 
in the past, permitting them to spend their time instead on evaluation of the implications of the trends 
for planning. Having solid and visual evidence of the various program trends also made it easier for 
faculty and administration alike to identify those programs that were weak in enrollments and in 
need of improvement or potential deletion. This helps make the program review process at least 
somewhat less threatening, and considerably less adversarial. 
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The ability to link Student Master Files and Master Schedule Files proved invaluable when we 
needed to analyze baccalaureate students' course-taking patterns and demographics. State legislation 
passed during the 1992 fiscal crisis mandated a differential fee of $40 per unit (in addition to the 
usual $10) for those community college students who already had a baccalaureate or higher degree. 
In order to plan for the Spring 1993 semester, when the new fee took effect, the district needed to 
answer two key questions: 

1 . What courses are these baccalaureate degree students enrolled in? 

2 . What do we know about these students demographically and in terms of their reasons 
for enrolling? 



To answer these questions, we analyzed the entire Fall 1992 Student Master File to determine the 
students' goals and demographics, then combined the information with the course information for 
these students contained on the Fall 1992 Master Schedule File. The resulting analysis provided a 
profile of the baccalaureate students districtwide: the majority were women, most over the age of 
30, predominantly white, who had enrolled primarily for occupational reasons. Each college was 
able to see its own baccalaureate student profile, and was provided an analysis of the courses these 
students were taking, including frequency of course enrollments by academic department. The 
colleges used this information to plan their Spring course schedules and to anticipate the areas which 
might be hardest hit by enrollment losses due to the fee. Finally, the list of baccalaureate students 
from the Student Master File was used as a mailing list for a special survey which was sent to each 
of these students to elicit additional information about their plans for the Spring 1993 semester and 
their reactions to the new baccalaureate fee. 

In Fall 1996, we again used this model to examine the "return" of the baccalaureate students after 
the differential fee was ended in January 1996, to help us better plan for the Spring 1997 semester. 

The Student Performance Data Base 



Perhaps the most critical addition to our research capabilities in terms of evaluating institutional 
effectiveness was the Student Performance Data Base. When we began to develop the SPDB, the 
district's student information was designed such that the demographic, application and transcript 
information was stored in a relational data base on the district's mainframe computer. The statistical 
package SPSS/Windows was installed on the college's research office personal computer. We knew 
that with demands for accountability increasing, we needed to look at data in new ways to answer 
the key questions on student preparation, persistence and performance. We decided to create our 
own relational data base at the college level that would link all the disparate files into a usable, 
effective structure. 

We needed to be able to answer the key questions about what was happening to students while they 
were in our college, whether they were persisting and performing successfully, and how performance 
might differ for different groups of students in different programs. To do such analyses we needed 
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the capability of essentially examining transcripts as would a counselor, looking not only at overall 
or course-specific GPA, but at the kinds of courses the student was taking remedial, transfer or 
occupational — since student preparation for college-level work is a key indicator of student 
performance, and an important issue in community colleges. 

Figure 5 illustrates the major components of the Student Performance Data Base. Essentially, 
student application, demographic and transcript data files on the mainframe computer were linked 
using student Social Security numbers and the file handling capabilities of a PC-based database 
package (FoxPro). Converting the transcript data files proved the most difficult, since the on-line 
viewing capabilities designed for the counselors at each college to view individual student transcripts 
meant that the relational data base structure was highly complex. Working with the district's 
research office, the college was able to have computerized transcript information downloaded to the 
college's research computer system. With the advancement of technology, larger disk storage media 
made it feasible to store vast amounts of data on a personal computer. The Student Performance 
Data Base now holds historical information on all students who attended Cosumries River College 
over the past thirteen years. 

In order to produce computerized transcript analyses for particular types of courses, we had to design 
a process to identify every course on the district's data base as to type — whether the course is a 
general education or an occupational education course, and at the transfer, AA degree or remedial 
level. In addition, we wanted to be able to select or aggregate the information at the department or 
even course level, so as to answer such questions as how students who take remedial English courses 
perform in courses at the Freshman Composition level, or what the GPAs of student athletes are 
when figured with and without PE courses. We could finally classify each course a student had 
attempted over the past thirteen years, taking into consideration renaming of courses and changes 
in classifications over time. Computerized transcript analysis was now a reality. 

Because the SPDB also includes the student application and demographic data, performance analyses 
can be done along demographic lines or according to student goal or major, and student populations 
can include any group of students identifiable by Social Security number — for example, those on 
financial aid, those in special programs or receiving special services. This permits true analyses of 
the effectiveness of programs and services on particular populations, or of pre- or co-requisites on 
student performance in subsequent courses. The information for specific populations in specific 
studies can be requested by staff at our college and downloaded in any of a variety of formats for use 
in subsequent analysis. Figure 6 shows a flowchart of the key elements of the Student Performance 
Data Base at the college level. 

The SPDB has been used in a wide variety of studies since its inception. Perhaps best-known is the 
district's Moving On: A Cooperative Study of Student Transfer (Jones, 1991), done with the 
University of California at Davis and with California State University at Sacramento. This study, 
which was published nationally, went far beyond the usual counts of transfer students to determine 
their levels of preparation, persistence and performance at the community college, and then how they 
performed in terms of units attempted and completed, GPA and baccalaureate degree attainment at 
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each of the universities. We were able to identify those students who had taken remedial courses 
in English, math or ESL at the community college level and then to track the remedial vs. non- 
remedial cohorts into and through the universities to determine the effects that remediation had on 
subsequent performance. Analyses were performed along demographic lines for various ethnic 
groups and for both genders. The findings that the community college transfers performed as well 
as or better than the university native students, and that the remedial cohort graduated from the 
universities in proportions not much different from the non-remedial cohorts were shared across 
all three segments and helped to identify profiles of the successful transfer student. This information 
has been used by counselors, transfer center directors and others to assist students in developing and 
reaching their transfer goals. 

Former Student Follow-Up Data Base 

The Los Rios district began its student follow-up efforts with a pilot survey in Spring 1983, long 
before the demand for accountability became an issue of state and national concern. We wanted to 
know what happened to our students as a result of their community college experience: Were they 
employed? Did they transfer to another college or university? Had their goals been met? (Figure 7). 

The Spring 1983 pilot study involved only "occupational" students, following guidelines required 
for Vocational Education Data System (VEDS) reporting. As a result of that study, Los Rios 
decided to expand its survey population to include all graduates and certificate earners (general 
education as well as occupational majors), plus the cohort of nonretuming nongraduates (referred 
to as NRSs) selected according to VEDS guidelines. Since that study also showed that the VEDS 
guidelines misidentified from 20% to 40% of the students' major programs, Los Rios decided to use 
its own record of the degrees/certificates actually awarded (if any). 

The Los Rios Student Follow-Up Data Base now includes responses from more than 15,000 former 
students who attended the district's three colleges between 1982 and 1996 (Figure 8). Survey results, 
reported in the district's Measures of Progress series, are aggregated on the basis of the respondents' 
self-reported educational goals: University Transfer, Occupational Preparation, Occupational 
Retraining, and Personal Interest, with a separate analysis of responses from NRSs (regardless of 
goal). The results are also summarized by program, by college, and for the district as a whole. 

The early Los Rios studies were conducted annually (1984 through 1987), and were compared in 
Measures of Progress: A Four-Year Retrospective (Lee, 1987). As recommended in that report, 
subsequent studies have been conducted biennially (but still include all students who would have 
been included in the annual surveys). Measures of Progress: Student Follow-Up, 1984-1991 (Lee, 
1991) discussed the series of studies in a newsletter format, identifying trends and summarizing the 
responses received during that eight-year period. Since that time the district's research office has 
repeated the studies, with the newest full report done in 1996. 

In addition to collecting the employment data required for reporting on occupational education 
students, the Los Rios questionnaire elicits information on the students' primary reason for enrolling 
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at the community college, their current educational status (including information on their transfer 
school, if any), and their ratings of community college courses completed and services used 
(Figure 9). Thus the surveys go far beyond "compliance" reporting and provide considerable 
additional information useful in college and district planning. 

Survey procedures have been modified over the thirteen-year period, usually in direct response to 
suggestions or requests from users of the data. One of the most important of these changes, made 
at the request of college deans of occupational information, was the development of program -specific 
summaries which are prepared for each major program having ten or more responses. Each 
summary shows the number and percentage of respondents answering each of the questionnaire 
items, as well as basic information regarding that program's survey population (number of graduates, 
certificate earners and NRSs; adjusted response rate; and demographic profiles of the respondents). 
This change of format and the breakdown of information by program makes the information much 
more useful to college staff in their curriculum and program review efforts and in accreditation self- 
study reports. 

Another refinement — the designing of individual skills-specific questionnaires for various 
occupational majors — resulted from Los Rios' involvement in the early stages of what is now called 
the Statewide Student Follow-Up System (SSFS): faculty at each of our colleges used drafts 
developed for that earlier project to design supplementary forms tailored specifically to Los Rios 
occupational programs they wanted to examine more closely. 

The Student Follow-Up Data Base enabled the district to comply in a timely manner with the 
Student Right-to-Know regulations, since we already had on hand a large amount of historical 
information regarding where former students found employment and the relationship of their jobs 
to their community college education. The data base was also very useful in meeting the V ATEA 
reporting requirements. 

Information from the data base has also been used in a variety of other ways: by the colleges, to 
provide information for their Occupational Education Advisory Committees; by members of Los 
Rios' Board of Trustees and the Director of Community and Media Relations, in responding to 
business/community concerns; in responding to the State Chancellor's Office accountability; and in 
conference presentations and published articles on community college accountability at both state 
and national levels. 

Los Rios continues to refine its student follow-up system with each new biennial survey, in order 
to continue to meet the need for even more comprehensive answers to the basic question: What 
happened to our students as a result of their community college experience? The enhancements 
described above and others in the planning stages will maintain the Student Follow-Up Data Base 
as a dynamic and essential component of the district's and the colleges' accountability models. 
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Measures of Progress 

Student Follow-Up Data: Key to Accountability 
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Beyond Compliance: Meets/Exceeds Requirements 
For SSFS, VATEA, Student Right-to-Know 



FROM ACCOUNTABILITY TO EFFECTIVENESS 



The genesis from the district's Student Flow Research Model to the new college-level CYA 
Institutional Effectiveness Model has taken more than a decade. Much trial and error has been 
involved, along with a strong focus on the original key questions from the Student Flow Research 
Model. Those questions are still the ones we need to answer, and our ability to examine our 
programs and services in terms of their effectiveness in producing successful students is now more 
important than ever. In this era of increasing student enrollment demand and diminishing resources, 
we no longer have the luxury of maintaining programs and services that are not meeting student 
needs. And more than ever before, we are being asked to demonstrate — to the Legislature and the 
Governor, to state and federal agencies, to our boards, to our communities, to our faculty and to our 
current and potential students — that what we are doing is working. 

In designing our model, we kept focused on the questions to be answered rather than on the data to 
be collected; that was true in the original Student Flow Research Model and it continues to be the 
best process for enhancing the model. Focusing on what happens to our students as they move into, 
through and beyond our colleges has helped us identify which pieces of the model we needed to add 
next— and helped us to work around seemingly impossible barriers creating what we needed as the 
technology permitted it (and sometimes when it didn't). As we were able to get to finer levels of 
analysis (course-taking patterns; the effects of remedial courses; GPA with and without PE courses), 
we have been better able to identify where our programs and services are having an impact and 
where they are not. Figure 10 summarizes the changes to the model over time, and Figure 1 1 
illustrates some of the key outcomes we have achieved — the move from compliance and 
accountability to institutional effectiveness. 

Perhaps most of all, we have learned how complex a process is analyzing institutional effectiveness 
and sometimes, how political. Documenting increases in the number of transfers a community 
college is sending to public four-year universities is one thing; finding out what proportion of those 
students ever received their baccalaureate degrees, and how it differed by gender and ethnic group, 
is quite another. While it is tempting to hold individual colleges accountable for particular outcomes 
(numbers who transfer, students who graduate), it is also true that colleges do not have control over 
all the factors that affect student performance, and that students themselves must take some 
responsibility for their success or failure. However, as researchers and educators who believe that 
the only real reason for research in the community colleges is to help improve our institutions on 
behalf of our students, and to determine which programs work best with which students under what 
circumstances, we believe that the more complete the information, the more compelling an argument 
for change. Moving from compliance and accountability to true analyses of institutional 
effectiveness is difficult and time-consuming. It may be the most we can do; it is the least our 
students deserve. 
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Student Flow Research Model 

“From Accountability to Effectiveness” 
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Key Question: 

Is What We’re Doing Working and How Do We Know? 
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